Image pickup unit and electronic apparatus

ABSTRACT

There is provided an image pickup unit capable of suppressing occurrence of false color and color mixture and acquiring a color image with high image quality. The image pickup unit includes: an image sensor including a plurality of pixels and acquiring an image pickup data; a variable filter provided on a light receiving face of the image sensor, and transmitting a selective wavelength; and a filter drive section (a wavelength selection circuit and a system control section) driving the variable filter and thereby setting its transmission wavelength. By acquiring the image pickup data while time-divisionally switching the transmission wavelength of the variable filter, pixel data corresponding to the transmission wavelength of the variable filter are acquired in a temporally-successive manner.

CROSS REFERENCES TO RELATED APPLICATIONS

The present application is a continuation of U.S. application Ser. No.14/347,132, filed on Mar. 25, 2014, which is a National Stage Entry ofInternational Application No. PCT/JP2012/075102 filed on Sep. 28, 2012,which claims priority to Japanese Priority Patent Application No. JP2011-221976 filed in the Japan Patent Office on Oct. 6, 2011, the entirecontent of which is hereby incorporated by reference.

BACKGROUND

The present disclosure relates to an image pickup unit that may bepreferable, for example, for color photography, and to an electronicapparatus including this.

Generally, in an imaging device (an image pickup unit) that shoots acolor image, a color filter that has a predetermined color arrangementis provided on an image sensor that includes a plurality of pixels, andthe color image is synthesized using image pickup data that aredispersed space-divisionally. Specifically, for example, a filter thatmay selectively transmit color light of one of R (red), G (green), and B(blue) is provided on the image sensor for each pixel. As a colorarrangement of R, G, and B in this case, for example, Bayer arrangementis typical.

On the other hand, as a measurement unit that utilizes spectroscopy, forexample, a measurement unit that may include, on an optical detectiondevice, for example, a variable filter that selectively transmits aspecific wavelength is used (for example, Patent Literatures 1 to 4). Asthe variable filter, a so-called liquid crystal Lyot filter can bementioned in which a transmission wavelength may be variable by voltagedrive, for example.

SUMMARY

Generally, an image pickup unit for color photography may use, forexample, a filter in which regions are separately painted with the useof respective pigments of R, G, and B as the above-described colorfilter, and allows color light of one of R, G, and B to be selectivelytransmitted in each region. Therefore, color light of only one color ofR, G, and B is allowed to be detected in one pixel. In particular, in animage corresponding to a subject that has a large contrast difference,degradation in image quality such as so-called false color and colormixture may be easily caused as a result of an image arithmeticprocessing.

Therefore, it is desirable to provide an image pickup unit capable ofsuppressing occurrence of false color and color mixture and acquiring acolor image with high image quality, and to provide an electronicapparatus that includes this.

An image pickup unit of an embodiment of the present disclosureincludes: an image pickup device including a plurality of pixels andoutputting an image pickup data; a variable filter provided on a lightreceiving face of the image pickup device, and configured to allow aselective transmission wavelength of incident light to be variable; anda filter drive section driving the variable filter and therebytime-divisionally switching the transmission wavelength.

An electronic apparatus of an embodiment of the present disclosureincludes the image pickup unit of the above-described embodiment of thepresent disclosure.

In the image pickup unit of an embodiment of the present disclosure, thefilter drive section drives the variable filter and time-divisionallyswitches its transmission wavelength. Accordingly, in the image pickupdevice, image data of colors corresponding to transmission wavelengthsof the variable filter are outputted in a temporally-successive manner.

According to the image pickup unit of an embodiment of the presentdisclosure, the filter drive section drives the variable filter, andtime-divisionally switches its transmission wavelength. Therefore, inthe image pickup device, it is possible to output, in atemporally-successive manner, the image data of colors corresponding tothe transmission wavelengths of the variable filter as the image pickupdata. By generating a color image based on such image pickup data, it ispossible to suppress occurrence of false color and color mixture, and toacquire a color image with high image quality.

Additional features and advantages are described herein, and will beapparent from the following Detailed Description and the figures.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a schematic diagram illustrating an outline configuration ofan image pickup unit according to an embodiment of the presentdisclosure.

FIG. 2 is a functional block diagram of the image pickup unit shown inFIG. 1.

FIG. 3 is an equivalent circuit diagram of a pixel circuit in an imagesensor.

FIG. 4 is an equivalent circuit diagram of a wavelength selectioncircuit shown in FIG. 1.

FIG. 5 is a cross-sectional diagram illustrating an outlineconfiguration of the image pickup unit shown in FIG. 1.

FIG. 6 is a schematic diagram of a cell unit that configures a variablefilter.

FIG. 7 (A) to (C) are schematic diagrams that each illustrate an exampleof a layout of the cell unit.

FIG. 8 is a conceptual diagram for explaining a function of the imagepickup unit shown in FIG. 1.

FIG. 9 is a timing diagram for driving the pixel circuit and thewavelength selection circuit.

FIG. 10 (A) to (C) are schematic diagrams each illustrating an exampleof time-divisional drive of the variable filter.

FIG. 11 is a flow chart illustrating an image processing operation.

FIG. 12 is a flow chart illustrating an image processing operationaccording to Modification 1.

FIG. 13 is a cross-sectional diagram illustrating an outlineconfiguration of an image pickup unit according to Modification 2.

FIG. 14 is a cross-sectional diagram illustrating an outlineconfiguration of a variable filter according to Modification 3.

FIG. 15 is an example of a combination of transmission wavelengths inrespective liquid crystal cells in the variable filter shown in FIG. 14.

FIG. 16 is a cross-sectional diagram illustrating an outlineconfiguration of the variable filter according to Modification 4.

FIG. 17 is a functional block diagram of an electronic apparatus(camera) according to an application example.

DETAILED DESCRIPTION

Hereinafter, some modes for carrying out the present disclosure will bedescribed with reference to the drawings. It is to be noted that thedescription will be given in the following order.

-   -   1. Embodiment (an example in which a transmission wavelength (R,        G, B) of a variable filter provided on an image sensor is        switched time-divisionally)    -   2. Modification 1 (another example of time-divisional drive for        R, G, B, and near-infrared rays, and of image processing)    -   3. Modification 2 (another example of the image sensor)    -   4. Modification 3 (another example of the variable filter (a        liquid crystal Lyot filter))    -   5. Modification 4 (an example in which a piezoelectric        Fabry-Perot interferometer is used for the variable filter)    -   6. Application Example (an example of an electronic apparatus)

Embodiment

[Configuration]

FIG. 1 illustrates a general configuration of an image pickup unit (animage pickup unit 1) according to an embodiment of the presentdisclosure. The image pickup unit 1 is an image pickup unit that issuitable for shooting a color image (a still image or a moving image).The image pickup unit 1 includes a variable filter 20 that is providedon a light receiving face of an image sensor 10 (image pickup device).The variable filter 20 is connected, via a wiring 10 a, to an electroniccircuit (a wavelength selection circuit 21 which will be describedlater) that is formed in the image sensor 10.

[Functional Block Configuration, Circuit Configuration]

FIG. 2 illustrates a functional block configuration of the image pickupunit 1. In the image pickup unit 1, the image sensor 10 includes a pixelarray section 12 on a substrate 11. Around this pixel array section 12,circuit sections (for example, a row scanning section 13, a horizontalselection section 14, a column scanning section 15, and a system controlsection 16) for driving the pixel array section 12 are arranged. Thevariable filter 20 is provided on the image sensor 10 to face the pixelarray section 12. The variable filter 20 is connected to the wavelengthselection circuit 21 formed on the substrate 10. The image pickup unit 1also includes an image processing section 22 on the substrate 11.

The image sensor 10 may be configured, for example, of a solid-stateimage pickup device such as a CMOS (Complementary Metal OxideSemiconductor). The pixel array section 12 is to be an image pickup areain the image pickup unit 1. In this pixel array section 12, pixels P(hereinafter, may be simply described as “pixel” in some cases) arearranged two-dimensionally in a matrix that each include a photoelectricconversion element (a photodiode 111A which will be described later)that generates a photoelectric charge of an electric charge amount inaccordance with a light amount of incident light and accumulates thegenerated photoelectric charge therein. For example, three wirings(specifically, a row selection line 171, a reset control line 172, and atransfer line 173 which will be described later) may be connected, aspixel drive lines 17, to the pixel P. The pixel drive lines 17 transmitdrive signals for reading signals. It is to be noted that, in FIG. 1,these pixel drive lines 17 are shown as one wiring for simplification.Further, a vertical signal line 18 for transmitting the read signals isfurther connected to the pixel P. A configuration of this pixel arraysection 12 will be described later.

The column scanning section 13 is configured of a shift resistor, anaddress decoder, etc. The column scanning section 13 is a pixel drivesection that may drive the respective pixels P in the pixel arraysection 12, for example, in a row unit basis. Signals outputted from therespective unit pixels in the pixel row that has been selectivelyscanned by the row scanning section 13 are supplied to the horizontalselection section 14 via the respective signal lines 18. The horizontalselection section 14 is configured of an amplifier, a horizontalselection switch, etc. that are provided for each vertical signal line18.

The column scanning section 15 is configured of a shift resister, anaddress decoder, etc. The column scanning section 15 sequentially drivesthe respective horizontal selection switches in the horizontal selectionsection 14 while scanning the respective horizontal selection switches.Through this selective scanning by the column scanning section 15, thesignals for the respective pixels that are transmitted via therespective vertical signal lines 18 are sequentially outputted to ahorizontal signal line 19, and are transmitted to outside of thesubstrate 11 via the horizontal signal line 19.

The variable filter 20 has a function as a so-called band pass filterthat selectively transmits light with a specific wavelength (forexample, any wavelength or wavelength band in a range from 0.3 μm to 2.5μm). A transmission wavelength of this variable filter 20 is allowed tobe switched by electric or mechanical control. In the presentembodiment, the image pickup unit 1 is used as an imaging device forcolor photography. Therefore, this variable filter 20 serves as a colorfilter that allows switching between the color light of three primarycolors of R, G, and B, and transmits the respective color light. Aspecific configuration of this variable filter 20 will be describedlater.

The wavelength selection circuit 21 is an electronic circuit for setting(selecting) the transmission wavelength of the above-described variablefilter 20. This wavelength selection circuit 21 may be controlled by thesystem control section 16 arranged in the image sensor 10. Thiswavelength selection circuit 21 may include, for example, transistors(transistors Tr5 to Tr7 which will be described later) for applyingsignal voltages to respective liquid crystal cells in theabove-described variable filter 20. Further, although detaileddescription will be given later, the wavelength selection circuit 21 maytime-divisionally switch the variable filter 20, for example, on a pixelregion unit basis, on a pixel column (row) unit basis, or on anall-pixel unit (plane unit) basis. A circuit configuration of thiswavelength selection circuit 21 will be described later. It is to benoted that this wavelength selection circuit 21 and the system controlsection 16 correspond to specific examples of the filter drive sectionin the present disclosure.

The image processing section 22 performs a predetermined arithmeticprocessing with the use of image pickup data acquired in the imagesensor 10, and may output, for example, color image data Dout. Aspecific image processing operation of this image processing section 22will be described later.

The system control section 16 receives a clock supplied from the outsideof the substrate 11, data that instructs an operation mode, etc., andcontrols operations of the image sensor 10, the wavelength selectioncircuit 21, and the image processing section 22. The system controlsection 16 may be configured, for example, of a microcomputer, etc.Specifically, the system control section 16 includes a timing generatorthat generates various timing signals. The system control section 16performs drive control of the row scanning section 13, the horizontalselection section 14, the column scanning section 15, and the wavelengthselection circuit 21, based on the timing signals generated by thetiming generator. Also, the system control section 16 outputs the imagepickup data from the horizontal signal line 19 to the image processingsection 22, and drives the image processing section 22.

It is to be noted that the circuit portions configured of the rowscanning section 13, the horizontal selection section 14, the columnscanning section 15, the wavelength selection circuit 21, and the systemcontrol section 16 may be formed directly on the substrate 11, or may bearranged on an outside control IC. Also, the circuit portion may beformed on another substrate connected by a cable, etc.

[Pixel Circuit]

FIG. 3 is an example of a circuit configuration of the pixel P thatincludes the photodiode 111A. The pixel P may include, for example, thephotodiode 111A (the photoelectric conversion element), four transistorsTr1, Tr2, Tr3, and Tr4, the above-described vertical signal line 18, andthe pixel drive lines 17 (the row selection line 171, the reset controlline 172, and the transfer line 173). As the pixel drive lines 17, forexample, three drive wirings that are the row selection line 171, thereset control line 172, and the transfer line 173 may be connected toeach pixel P in such a manner that the three drive wirings are common tothe respective pixels in the same pixel row. Each of one ends of the rowselection line 171, the reset control line 172, and the transfer line173 is connected to an output end corresponding to each pixel row of therow scanning section 13 in a pixel row unit basis.

Each of the transistors Tr1 to Tr4 may be, for example, anN-channel-type field effect transistor, and may be configured using, forexample, a silicon-based semiconductor such as microcrystalline silicon,crystalline silicon, or polycrystalline silicon. Alternatively, oxidesemiconductor such as indium-gallium-zinc oxide (InGaZnO) or zinc oxide(ZnO) may be used.

The transistor Tr1 is a transfer transistor. A gate of the transistorTr1 is connected to the transfer line 173, one end of a source and adrain thereof is connected to one end (for example, a cathode) of thephotodiode 111A, and the other end thereof is connected to an FD (afloating diffusion). A transfer pulse φ TRF which is active at a highlevel (for example, a Vdd level) (hereinafter, described as“high-active”) is supplied to the gate of the transistor Tr1 via thetransfer line 173. Accordingly, the transistor Tr1 becomes an ON state,and the transistor Tr1 transfers, to the FD, the photoelectric chargethat has been subjected to photoelectric conversion in the photodiode111A.

The transistor Tr2 is a reset transistor. A gate of the transistor Tr2is connected to the reset control line 172, one end of a source and adrain thereof is connected to a pixel power source Vdd, and the otherend thereof is connected to the FD. A high-active reset pulse φRST issupplied to the gate of the transistor Tr2 via the reset control line172. Accordingly, the transistor Tr2 becomes an ON state, and thetransistor Tr2 resets the FD by discarding the electric charge that hasbeen accumulated in the FD to the pixel power source Vdd, before thetransfer of the signal electric charge from the photodiode 111A to theFD.

The transistor Tr3 is an amplifier transistor. A gate of the transistorTr3 is connected to the FD, one end of a source and a drain thereof isconnected to the pixel power source Vdd, and the other end thereof isconnected to the transistor Tr4. This transistor Tr3 outputs an electricpotential of the FD after being reset as a reset signal (a reset level)Vreset. The transistor Tr3 also outputs an electric potential of the FDafter the transfer of the signal electric charge as a light accumulationsignal (a signal level) Vsig.

The transistor Tr4 is a selection transistor. A gate of the transistorTr4 is connected to the row selection line 171, one of a source and adrain thereof is connected to the transistor Tr3, and the other endthereof is connected to the vertical signal line 18. A high-activeselection pulse φSEL is supplied to the gate of the transistor Tr4 viathe row selection line 171. Accordingly, the transistor Tr4 becomes anON state, and relays a signal outputted from the transistor Tr3 to thevertical signal line 18 while allowing the pixel P to be in a selectedstate. It is to be noted that this transistor Tr4 may be connectedbetween the pixel power source Vdd and the drain of the transistor Tr3.Further, the transistor Tr3 may also serve as the transistor Tr4. Inother words, a circuit configuration that has three transistors may beadopted.

One end (for example, an anode) of the photodiode 111A may be connected,for example, to ground (GND). The photodiode 111A converts receivedlight into a photoelectric charge of an electric charge amountcorresponding to the amount of the received light. The other end (forexample, the cathode) of the photodiode 111A is electrically connectedto the gate of the transistor Tr3 via the transistor Tr1 and the FD. Asensitivity range (a received light wavelength band) of this photodiode111A may be in a range, for example, from 0.3 μm to 20 μm. Thisphotodiode 111A is allowed to perform photoelectric conversion on awavelength in such a range. This photodiode 111A generates a signalelectric charge of an electric charge amount in accordance with a lightamount (a received light amount) of incident light by receiving areference electric potential at the cathode side thereof. In the presentembodiment, as will be described later, the variable filter 20selectively transmits light of three wavelengths of R (red: for example,from 620 nm to 750 nm), G (green: for example, from 495 nm to 570 nm),and B (blue: for example, from 450 nm to 495 nm), and a color image isgenerated based on pixel data corresponding to the respective colorlight. Therefore, it is enough for the photodiode 111A to be sensitiveto a wavelength (for example, about 380 nm to 750 nm) corresponding tovisible light rays that include wavelengths of R, G, and B. As such aphotoelectric conversion material, for example, polycrystalline silicon,crystalline silicon, microcrystalline silicon, amorphous silicon, etc.can be mentioned.

[Wavelength Selection Circuit 21]

FIG. 4 is an example of a circuit configuration of the wavelengthselection circuit 21. The wavelength selection circuit 21 includes atransistor Tr and an electronic circuit. The transistor Tr is forapplying a signal voltage Vout to a liquid crystal cell LC thatconfigures the variable filter 20. The electronic circuit is configuredof an operational amplifier 28 and a digital potentiometer 29 that arefor setting the signal voltage Vout. A gate of the transistor Tr isconnected to a selection line 27 for transmitting a timing controlsignal from the system control section 16. One of a source and a drainof the transistor Tr is connected to an output terminal of theoperational amplifier 28, and the other thereof is connected to anelectrode (an electrode 23 a 1, 23 b 1, or 23 c 1 which will bedescribed later) of the liquid crystal cell LC. It is to be noted that,as will be described later, in the present embodiment, the variablefilter 20 includes three liquid crystal cells 20A to 20C. Thetransmission wavelength is time-divisionally set (switched) inaccordance with a combination of signal voltages φ1, φ2, and φ3 that areseparately applied to these liquid crystal cells 20A to 20C. Therefore,the above-described circuit is provided for each of the liquid crystalcells 20A to 20C (or, for each cell unit U which will be describedlater), and the corresponding transistors Tr5 to Tr7 applies the signalvoltages φ1, φ2, and φ3 at selective timings. It is to be noted that, inFIG. 4, the liquid crystal cell LC corresponds to one of the liquidcrystal cells 20A to 20C, the transistor Tr corresponds to one of thetransistors Tr5 to Tr7, and the signal voltage Vout corresponds to oneof the signal voltages φ1, φ2, and φ3.

It is to be noted that, in this example, an arbitrary signal voltageVout is allowed to be set by digital signal control using the digitalpotentiometer 29. In a case having such a configuration, it is possibleto perform correction in correspondence with characteristic variationsat the time of manufacturing by adjusting the signal voltage Vout.Alternatively, it is possible to set the transmission wavelength at thetime of shooting to an arbitrary wavelength. For example, it may bepossible to switch the transmission wavelength at the time of shootingto a wavelength (such as infrared rays or ultraviolet rays) other thanR, G, and B. Further, in this example, a voltage of output of thedigital potentiometer 29 is adjusted by the operational amplifier 28,and this is set as the signal voltage Vout. However, the operationalamplifier 28 and the digital potentiometer 29 may be provided asnecessary. For example, when the transmission wavelength of the variablefilter 20 is set to a fixed value, and application thereof is alsolimited, it is enough that the signal voltage Vout is a fixed value.Therefore, in this case, it is possible to adopt a circuit configurationin which the above-described digital potentiometer 29 is not provided.Further, when voltages higher than the voltage supplied by the systemcontrol section 16 are necessary as the signal voltages φ1, φ2, and φ3,a high voltage power source may be provided separately, or the highervoltages may be supplied from outside. Alternatively, amplificationeffect may be improved by using output from a charge pump circuit, etc.as a power source VDD2 of the operational amplifier 28.

[Cross-Sectional Configuration]

FIG. 5 schematically illustrates part of (around boundary of the pixelarray section 12 and its peripheral sections) a cross-sectionalstructure of the image pickup unit 1. The image sensor 10 may be, forexample, a CMOS (Complementary Metal Oxide Semiconductor) of a so-calledback illumination type that may include a device layer 113 on thesubstrate 11, and in a layer located in an upper position compared to awiring layer 112. The device layer 113 includes a photoelectricconversion layer 114. In this structure, for example, theabove-described transistors Tr1 to Tr4 for pixel driving and thetransistors Tr5 to Tr7 configuring part of the wavelength selectioncircuit 21 are formed in the device layer 113 together with thephotoelectric conversion layer 114. An on-chip lens (OCL) for lightcondensing is provided directly above the photoelectric conversion layer114. In the pixel array section 12, for each pixel P, the photodiode111A that includes the photoelectric conversion layer 114 and theon-chip lens 115, and the above-described four transistors Tr1 to Tr4are provided. The variable filter 20 is laminated on a light receivingface side of this image sensor 10.

The variable filter 20 may be configured, for example, of a so-calledliquid crystal Lyot filter. The liquid crystal Lyot filter is allowed toachieve maximum transmittance of almost 100% and a full width at halfmaximum (FWHM) of about 100 nm by a combination of liquid crystal cellsand polarizing plates as will be described later. Therefore, the liquidcrystal Lyot filter has a sufficient performance as a color filter. Itis to be noted that, in this liquid crystal Lyot filter, a plurality ofliquid crystal cells are driven with a voltage to set the transmissionwavelength as will be described later. Therefore, response speed of theliquid crystal easily influences image quality. Therefore, a liquidcrystal cell that has response speed of about 5 ms or smaller may bepreferably used as the liquid crystal cell. When the response speed is 5ms or smaller, it is possible to sufficiently follow 60 frame/second ofactual moving image shooting. Therefore, it is possible to suppressdegradation in image quality resulting from delay in response. Such aliquid crystal Lyot filter may be, for example, a filter in which aplurality of liquid crystal cells that are allowed to be electricallydriven in a separated manner and a plurality of polarizing plates arelaminated alternately. In the variable filter 20, the transmissionwavelength of the variable filter 20 as a whole varies in accordancewith a combination of drive voltages that are applied to such aplurality of liquid crystal cells.

Such a variable filter 20 is connected to the electronic circuit (thewavelength selection circuit 21) formed in the image sensor 10 via ananisotropic conductive film 210 and a connection bump 211. It is to benoted that the anisotropic conductive film 210 and the connection bump211 correspond to specific examples of the wiring 10 a in FIG. 1. Thisvariable filter 20 may be configured, for example, of the liquid crystalLyot filter as described above, and is a filter in which the pluralityof (here, three) liquid crystal cells 20A, 20B, and 20C, and polarizingplates 22 a, 22 b, 22 c, and 22 d are laminated alternately. In such avariable filter 20, the polarizing plates 22 a to 22 d may be arranged,for example, so as to allow absorption axes thereof to coincide with oneanother. However, as with a so-called modification-type Lyot filter, theabsorption axis of the polarizing plate 22 d on a light emitting side(on an image sensor 10 side) may be arranged to be orthogonal to theabsorption axes of other polarizing plates 22 a to 22 c (may be arrangedto form a crossed-Nicols arrangement).

The liquid crystal cells 20A to 20C each transmit a selective wavelengthusing ECB (Electrically Controlled Birefringence) of liquid crystal. Inother words, only specific wavelengths are transmitted by the variablefilter 20 as a whole in accordance with a combination of thetransmission wavelengths of the respective plurality of liquid crystalcells 20A to 20C. The transmission wavelength of the variable filter 20is allowed to have various bands depending on the number of stages (thenumber of laminated layers) of the liquid crystal cells to beconfigured, a combination of drive voltages applied thereto, etc. Also,it is possible to allow a transmission spectrum of the transmissionwavelength to be sharp (to narrow the band of the transmissionwavelength), or in reverse, to allow the transmission spectrum to begentle (to widen the band of the transmission wavelength) dependingthereon.

In each of these liquid crystal cells 20A to 20C, a liquid crystal layer24 is sealed between a pair of transparent substrates 26 a and 26 b. Theliquid crystal cell 20A includes electrodes 23 a 1 and 23 a 2 configuredof transparent conductive films of ITO (indium-tin oxide) or the like ona liquid crystal layer 24 side of the transparent substrates 26 a and 26b. A predetermined voltage is applied to the liquid crystal layer 24through these electrodes 23 a 1 and 23 a 2. One electrode 23 a 1 of theelectrodes 23 a 1 and 23 a 2 is connected to the transistor Tr5 via theanisotropic conductive film 210 and the connection bump 211. The otherelectrode 23 a 2 may be connected, for example, to the ground.Accordingly, the predetermined drive voltage φ1 is applied to the liquidcrystal cell 20A via the transistor Tr5. Similarly, also in the liquidcrystal cell 20B, electrodes 23 b 1 and 23 b 2 for applying a voltage tothe liquid crystal layer 24 are provided on faces on the liquid crystallayer 24 side of the transparent substrates 26 a and 26 b. One electrode23 b 1 thereof is connected to the transistor Tr6 via the anisotropicconductive film 210 and the connection bump 211, and the other electrode23 b 2 may be connected, for example, to the ground. Accordingly, thepredetermined drive voltage φ2 is applied to the liquid crystal cell20B. Similarly, also in the liquid crystal cell 20C, electrodes 23 c 1and 23 c 2 for applying a voltage to the liquid crystal layer 24 areprovided on faces on the liquid crystal layer 24 side of the transparentsubstrates 26 a and 26 b. One electrode 23 c 1 thereof is connected tothe transistor Tr7 via the anisotropic conductive film 210 and theconnection bump 211, and the other electrode 23 c 2 may be connected,for example, to the ground. Accordingly, the predetermined drive voltageφ3 is applied to the liquid crystal cell 20C.

It is to be noted that, in the liquid crystal cells 20A to 20C, each ofthe substrates 26 a and 26 b extends to a peripheral region of the pixelarray section 12. Through-hole vias of the number (here, three) of theliquid crystal cells 20A to 20C are formed to run through in a thicknessdirection in the peripheral region. In the present embodiment, thesethrough-hole vias, the anisotropic conductive film 210, and theconnection bump 211 are used to secure electric connection between eachof the one electrodes 23 a 1, 23 b 1, and 23 c 1 of the liquid crystalcells 20A to 20C and each of the corresponding transistors Try to Tr7.Further, the other electrodes 23 a 2, 23 b 2, and 23 c 2 of the liquidcrystal cells 20A to 20C are each held at a common ground electricpotential. Therefore, the other electrodes 23 a 2, 23 b 2, and 23 c 2are connected to a GND line arranged in the image sensor 10 with the useof the through-hole via common to the respective liquid crystal cells,the anisotropic conductive film 210, and the connection bump 211.However, a technique to electrically connect the respective liquidcrystal cells 20A to 20C to the electronic circuit arranged in the imagesensor 10 is not limited to that described above, and various techniquessuch as a technique using another thorough-hole via and bonding may beused.

Due to the above-described configuration, in the variable filter 20, thetransmission wavelength is allowed to be set (switched) based on thecombination of the drive voltages φ1 to φ3 to be applied to therespective liquid crystal cells 20A to 20C. In the image pickup unit 1of the present embodiment, the transmission wavelength of this variablefilter 20 is time-divisionally (or, time-divisionally andspace-divisionally) switched between the respective wavelengths of R, G,and B, and light is received in the image sensor 10.

This variable filter 20 has the structure in which the plurality ofliquid crystal cells that are allowed to be electrically controlled in aseparated manner are laminated as described above. However, in such alaminated cell structure, the variable filter 20 may be configured toallow one transmission wavelength to be set in common to all pixels, orthe variable filter 20 may include a plurality of sub-filters in orderto set the transmission wavelengths separately for the respectivepixels. Specifically, the variable filter 20 is configured to includeone or a plurality of cell units (sub-filters) U each of which isconfigured of the liquid crystal cells 20A to 20C and the polarizingplates 22 a to 22 d as shown in FIG. 6. In other words, the variablefilter 20 is configured of one cell unit U, or has a plurality of cellunits U to which voltages are allowed to be applied separately. When thevariable filter 20 has the plurality of cell units U, it is enough thatat least the electrodes 23 a 1, 23 b 1, and 23 c 1 are divided into aplurality of pieces (for each pixel row (column) or for each pixel asshown below as examples). Other electrodes 23 a 2, 23 b 2, and 23 c 2,the substrates 26 a and 26 b, the liquid crystal layer 24, and thepolarizing plates 22 a to 22 d may be provided to be common to all ofthe pixels P, or may be divided. In this case, a region, in thelamination structure of the variable filter 20, corresponding to each ofthe divided electrodes corresponds to the above-described cell unit U.

For example, as shown in (A) of FIG. 7, one cell unit U (in this case,the same as the variable filter 20) may be provided over the entire face(all of the pixels P) of the pixel array section 12. In this case, thetransmission wavelength is allowed to be switched collectively for theface for each image pickup frame (in a time-divisional manner). Thus, afull-color image is allowed to be generated with a relatively-simplearithmetic processing. However, in a case of this face-collectivecontrol, a transfer rate may be desirably high-speed. In this case, forexample, a technology of achieving high transfer rate of 34.8 Gbps(ISSCC2011: 23.11 “a17.7 Mpixel 120 fps CMOS Image Sensor with 34.8 Gb/sReadout”, T. Toyama et al., Sony Corp. and Sony LSI Corp.), etc. areallowed to be used.

Alternatively, as shown in (B) of FIG. 7, the cell units U that eachface a pixel column (or a pixel row) in the pixel array section 12 maybe provided. In this case, it is possible to time-divisionally switchthe transmission wavelength, and to switch the transmission wavelengthfor each pixel column also in one frame (in a space-divisional manner).A high transfer rate is not necessary in such control compared to theabove-described case of the surface-collective control, and a colorimage is allowed to be generated easily by an arithmetic operationbetween adjacent pixels.

Moreover, as shown in (C) of FIG. 7, the cell units U that each face apixel P in the pixel array section 12 may be provided. In this case, itis possible to time-divisionally switch the transmission wavelength, andto switch the transmission wavelength for each pixel P also in one frame(in a space-divisional manner). Accordingly, as with the above-describedexample shown in (B) of FIG. 7, there are advantages that the transferrate is allowed to be relatively low and that a color image is easilygenerated. Further, this also allows use for shooting only of aselective region with a specific wavelength. Accordingly, it is possibleto improve contrast by acquiring information only on a specific object,or by removing a specific component such as a reflection component.

[Functions and Effects]

Functions and effects of the present embodiment will be described withreference to FIG. 2 to FIG. 11. FIG. 8 is a conceptual diagram forexplaining an operation of acquiring image pickup data by the variablefilter 20 and the image sensor 10 in the present embodiment.

In the image pickup unit 1, the system control section 16 drives the rowscanning section 13, the horizontal selection section 14, and the columnscanning section 15 in the image sensor 10, and thereby, exposure isperformed in each pixel P (the photodiode 111A), and by photoelectricconversion, an electric signal of an electric charge amount inaccordance with the amount of the received light is obtained in thepixel array section 12. The obtained electric signal isline-sequentially read out to the vertical signal line 18, and then, isoutputted to the image processing section 22 as image pickup data viathe horizontal signal line 19. The image processing section 22 performsa predetermined image arithmetic processing based on the inputted imagepickup data, and outputs image data Dout corresponding to a color image.

In the present embodiment, at the time of the above-described imagepickup operation in the image sensor 10, the system control section 16drives the wavelength selection circuit 21, and the transmissionwavelength of the variable filter 20 is set. Accordingly, only aspecific wavelength out of wavelengths that have entered the variablefilter 20 from the subject side passes through the variable filter 20,and enters the pixel array section 12 in the image sensor 10.

[Setting of Transmission Wavelength of Variable Filter 20]

Here, setting of the transmission wavelength of the variable filter 20is performed as follows. Specifically, as shown in FIG. 8, the systemcontrol section 16 applies, with the use of the wavelength selectioncircuit 21, different voltages to the respective liquid crystal cells20A to 20C that configure the variable filter 20. For example, thesystem control section 16 may apply the drive voltage φ1 to the liquidcrystal cell 20A, may apply the drive voltage φ2 to the liquid crystalcell 20B, and may apply the drive voltage φ3 to the liquid crystal cell20C. Depending on the combination of the drive voltages φ1 to φ3 (thecombination of the transmission wavelengths of the respective liquidcrystal cells 20A to 20C) at this time, only a specific wavelength istransmitted by the variable filter 20 as a whole, and is emitted to theimage sensor 10 side. In other words, an image of the subject isacquired as image pickup data for the specific wavelength selected withthe use of the variable filter 20. Specifically, as shown in FIG. 8,depending on the combination of the drive voltages φ1 to φ3, thevariable filter 20 selectively transmits red light Lr, and thereby,one-color data of R (pixel data Dr) is obtained in each pixel P.Alternatively, one-color data of G (pixel data Dg) is obtained byselectively transmitting green light Lg. Alternatively, one-color dataof B (pixel data Db) is obtained by selectively transmitting blue lightLb.

(A) to (E) of FIG. 9 illustrate an example of a timing of drive, by thesystem control section 16, of the pixel circuit and the wavelengthselection circuit. In these, (A) of FIG. 9 illustrates the reset pulseφRST of the transistor Tr2 (the reset transistor). R1, R2, . . . ,illustrate pulse waveforms in the respective reset control lines 172provided for the respective pixel rows (or pixel columns). (B) of FIG. 9illustrates the transfer pulse φTRF of the transistor Tr1 (the transfertransistor). T1, T2, . . . , illustrate pulse waveforms in therespective transfer lines 173 provided for the respective pixel rows.(C) of FIG. 9 illustrates the selection pulse φSEL of the transistor Tr4(the selection transistor). S1, S2, . . . , illustrate pulse waveformsin the respective row selection lines 171 provided for the respectivepixel rows. On the other hand, (D) of FIG. 9 illustrates the drivevoltages φ1 to φ3 for the respective liquid crystal cells 20A to 20C inthe variable filter 20.

As described above, the system control section 16 drives the rowscanning section 13, the horizontal selection section 14, and the columnscanning section 15 in the image sensor 10 and the wavelength selectioncircuit 21 in synchronization with one another. Accordingly, as shown in(E) of FIG. 9, signal electric charges sig1 y, sig2 y, . . . areacquired for the respective vertical signal lines 18 as one-color datacorresponding to the combination of the drive voltages φ1 to φ3.

[Time-Divisional Switching Operation of Transmission Wavelength (R, G,and B)]

However, in the present embodiment, light is received in the imagesensor 10 while the transmission wavelength of the variable filter 20 isswitched time-divisionally, and thereby, image pickup data is acquired.Specifically, as schematically shown in (A) to (C) of FIG. 10, thevariable filter 20 is driven to allow the respective color light of R,G, and B are transmitted in a time-divisional switching manner, andone-color data (pixel data) of R, G, and B are acquired in atemporally-successive manner in each pixel P in the image sensor 10.

At this time, the switching of the transmission wavelength of thevariable filter 20 is performed separately for the above-describedrespective one or the plurality of cell units U shown in (A) to (C) ofFIG. 7. For example, as in the example in (A) of FIG. 7, when one cellunit U is provided over the entire face (all of the pixels P) in thepixel array section 12, the transmission wavelength is time-divisionallyswitched collectively for all of the pixels (collectively for the face).Specifically, as shown in (A) of FIG. 10, R light is selectivelytransmitted to acquire one-color pixel data of R at a timing t1, G lightis selectively transmitted to acquire one-color pixel data of G at atiming t2 subsequent to the timing t1, and B light is selectivelytransmitted to acquire one-color pixel data of B at a timing t3subsequent to the timing t2.

Alternatively, as in the example in (B) of FIG. 7, when the variablefilter 20 includes the cell units U (U1, U2, U3, . . . ) that each facea pixel column (or a pixel row) in the pixel array section 12, inaddition to the above-described time-divisional control, thetransmission wavelengths may be repeatedly controlled in aspace-divisional manner to be different between the respective cellunits U1, U2, U3, . . . . Specifically, as shown in (B) of FIG. 10, at atiming t1, the R light, the G light, and the B light are transmitted inthe cell unit U1, the cell unit U2, and the cell unit U3, respectively.Further, at a timing t2 subsequent to this timing t1, switching isperformed to allow the G light, the B light, and the R light to betransmitted in the cell unit U1, the cell unit U2, and the cell unit U3,respectively. Further, at a timing t3 subsequent to the timing t2,switching is performed to allow the B light, the R light, and the Glight to be transmitted by the cell unit U1, the cell unit U2, and thecell unit U3, respectively.

Alternatively, as in the example in (C) of FIG. 7, when the variablefilter 20 includes the cell units U (U1, U2, U3, U4, . . . ) that eachface a pixel P in the pixel array section 12, in addition to theabove-described time-divisional control, the transmission wavelengthsmay be repeatedly controlled in a space-divisional manner to bedifferent between the respective cell units U1, U2, U3, U4, . . . .Specifically, as shown in (C) of FIG. 10, at a timing t1, the R light,the G light, the G light, and the B light are transmitted in the cellunit U1, the cell unit U2, the cell unit U3, and the cell unit U4,respectively. Further, at a timing t2 subsequent to this timing t1,switching is performed to allow the G light, the B light, the B light,and the R light to be transmitted in the cell unit U1, the cell unit U2,the cell unit U3, and the cell unit U4, respectively. Further, at atiming t3 subsequent to the timing t2, switching is performed to allowthe B light, the R light, the R light, and the G light to be transmittedin the cell unit U1, the cell unit U2, the cell unit U3, and the cellunit U4, respectively.

It is to be noted that, as in the above-described (B) and (C) of FIG.10, when the transmission wavelength is controlled to be switched alsospace-divisionally, the combination of the drive voltages φ1 to φ3 maybe varied for each cell unit U, and the respective cell units U may bedriven separately with the use of a voltage.

By the above-described time-divisional switching control of thetransmission wavelength of the variable filter 20, in the image sensor10, the respective one-color data (pixel data Dr, Dg, and Db) of R, G,and B is successively acquired in time series in each pixel P. Imagepickup data D0 that includes the time-series data of R, G, and Bacquired in such a manner is outputted to the image processing section22 as described above.

[Image Processing Operation]

FIG. 11 is a flow chart illustrating an example of an image arithmeticprocessing operation in the image processing section 22. The imagepickup data D0 includes pixel data of the respective colors of R, G, andB that are acquired in a temporally-successive manner as describedabove. The image processing section 22 generates a color image of threeprimary colors by an arithmetic processing using such image pickup dataD0.

Specifically, first, the image processing section 22 acquires one-colorpixel data Dr(t1), Dg(t2), and Db(t3) of R, G, and B obtained atsuccessive timings t1 to t3 (corresponding to the above-describedtimings t1 to t3 in (A) of FIG. 7) as the image pickup data D0 based onthe control by the system control section 16 (step S1).

Subsequently, the image processing section 22 uses the acquired pixeldata Dr(t1), Dg(t2), and Db(t3) to estimate a fine displacement inposition of the subject between temporally-successive image pickupframes at an accuracy of sub-pixel (registration), and generates pixeldata of three colors of R, G, and B at the timing t1 (step S2). This isa processing to fill (interpolate) a gap between pixels with the use ofthe pixel data between temporally-successive frames, assuming that aninterval Δt between the respective timings t1 to t3 is sufficientlyshort. In particular, in the case of shooting a moving image, eachtiming interval is sufficiently short as, for example, about severaltens ms. Therefore, it is allowed to be assumed that a variation in theamount of received light is resulting only from parallel movement of thesubject. As a registration technique, for example, an optical flowmethod as described below, etc. may be used.

In the optical flow method, a movement amount of the subject in a fineperiod is determined, and an image is generated by interpolation basedon this movement amount. Here, in x- and y-directions that areorthogonal to each other in a two-dimensional plane, when a movementamount in the x-direction is expressed as u, and a movement amount inthe y-direction is expressed as v, the respective movement amounts u andv are allowed to be determined by minimizing an error function E (u, v)shown in the following Expression (1). It is to be noted that, inExpression (1), I₁ represents an image observed at a timing t, and I₂represents an image observed at a timing at which Δt has elapsed fromthe timing t. When both of the movement amounts u and v are extremelysmall, I₂ is allowed to be approximated as the following Expression (2)by Taylor series expansion. Accordingly, the error function E (u, v) isexpressed as the following Expression (3). Further, this Expression (3)is partially differentiated for each of u and v, and u and v that allowthe partial differentiation to be 0 are determined, which are expressedas the following Expression (4). It is to be noted that, in Expression(4), A and b are expressed by Expression (5) and Expression (6),respectively.

$\begin{matrix}{{E\left( {u,v} \right)} = {\sum\limits_{x,y}\;\left( {{I_{2}\left( {{x + u},{y + v}} \right)} - {I_{1}\left( {x,y} \right)}} \right)^{2}}} & (1) \\{{I_{2}\left( {{x + u},{y + v}} \right)} \approx {{I_{2}\left( {x,y} \right)} + {u\frac{\partial{I_{2}\left( {x,y} \right)}}{\partial x}} + {v\frac{\partial{I_{2}\left( {x,y} \right)}}{\partial y}}}} & (2) \\{{E\left( {u,v} \right)} = {\sum\limits_{x,y}\;\left( {{I_{2}\left( {x,y} \right)} + {u\frac{\partial{I_{2}\left( {x,y} \right)}}{\partial x}} + {v\frac{\partial{I_{2}\left( {x,y} \right)}}{\partial y}} - {I_{1}\left( {x,y} \right)}} \right)^{2}}} & (3) \\{\begin{pmatrix}u \\v\end{pmatrix} = {A^{- 1}b}} & (4) \\{A = {\sum\limits_{x,y}\;{{\nabla{I_{2}\left( {x,y} \right)}} \cdot {\nabla\;{I_{2}\left( {x,y} \right)}^{T}}}}} & (5) \\{b = {- {\sum\limits_{x,y}\;{{\nabla{I_{2}\left( {x,y} \right)}}\left( {{I_{2}\left( {x,y} \right)} - {I_{1}\left( {x,y} \right)}} \right)}}}} & (6)\end{matrix}$

At the time of the registration in step S2, the image processing section22 first uses the above-described optical flow method to calculate amovement amount (u12, v12) between the timing t1 at which the R pixeldata Dr(t1) is acquired and the timing t2 at which the G pixel dataDg(t2) is acquired (step S21). Taking into consideration the movementamount (u12, v12) calculated in such a manner, G pixel data Drg(t1) atthe timing t1 is estimated (step S22). Also, on the other hand, theimage processing section 22 uses the above-described optical flow methodto calculate a movement amount (u13, v13) between the timing t1 at whichthe R pixel data Dr(t1) is acquired and the timing t3 at which the Bpixel data Db(t3) is acquired (step S23). Taking into consideration themovement amount (u13, v13) calculated in such a manner, B pixel dataDrb(t1) at the timing t1 is estimated (step S24).

Subsequently, the image processing section 22 adds the G pixel dataDrg(t1) and the B pixel data Drb(t1) at the timing t1 determined by theabove-described estimation arithmetic processing to the R pixel dataDr(t1) acquired at the timing t1. The image processing section 22thereby generates color (thee-primary-color) pixel data (data for onepixel) Drgb(t1) at the timing t1. Further, the image processing section22 performs such an arithmetic processing on all of the pixels P, andsums them up. Thus, a color image Irgb(t1) that has been shot at thetiming t1, and has a resolution corresponding to the number N of thepixels in the pixel array section 12 is generated. Alternatively, thecolor image Irgb(t1) may be generated by the following procedure.Specifically, the pixel data Dr(t1), Drg(t1), and Drb(t1) at the timingt1 are determined by the above-described registration for all of thepixels P, and the determined data are synthesized for each color. Thus,an R image Ir(t1), a G image Ig(t1), and a B image Ib(t1) are generated.Further, these images Ir(t1), Ig(t1), and Ib(t1) of the respectivecolors may be summed up to generate the color image Irgb(t1).

Further, pixel data are acquired in a temporally-successive mannerrepeatedly in order of R→G→B also for a timing t4 and the subsequenttimings, and similar arithmetic processings are performed using therespective timings t2, t3, . . . , as references. Thus, color imagesIrgb(t2), Irgb(t3), . . . are allowed to be generated also at the timingt2 and the subsequent timings. For example, color pixel data Drgb(t2) atthe timing t2 may be determined as follows. Specifically, a movementamount (u23, v23) between the timing t2 at which the G pixel data Dg(t2)is acquired and the timing t3 at which the B pixel data Db(t3) isacquired is calculated. Taking into consideration this movement amount(u23, v23), B pixel data Dgb(t2) at the timing t2 is estimated. On theother hand, a movement amount (u24, v24) between the timing t2 at whichthe G pixel data Dg(t2) is acquired and the timing t4 at which the Rpixel data Dr(t4) is acquired is calculated. Taking into considerationthis movement amount (u24, v24), R pixel data Dgr(t2) at the timing t2is estimated. Thus, the B pixel data Dgb(t2) and the R pixel dataDgr(t2) at the timing t2 are generated.

It is to be noted that, in the above-described arithmetic processing,registration and RGB addition are performed for each pixel data obtainedfrom the pixel P, and the resulted data are synthesized at last togenerate a color image. However, the registration is not performedlimitedly on a pixel unit basis, and the above-described registrationmay be performed on a block region unit basis, on a pixel column unitbasis, a pixel row unit basis, or on an all-pixel (face, frame) unitbasis to generate a color image data. The block region is configured oftwo or more pixels. In other words, as long as color images for therespective timings are generated with the use of the time-series data ofR, G, and B, it is possible to obtain an effect (of suppressingoccurrence of false color, color mixture, etc.) which will be describedlater.

Further, in a case of shooting a still image in particular, a colorimage may be also generated as follows when the movement amount of thesubject is extremely small (for example, an image is not shifted for aperiod of about 50 ms). Specifically, in a case where it is assumed thatthe time interval of the timings t1 to t3 is sufficiently short, in apixel P, an incident light spectrum A (λ) of the timing t (=Σt_(i)/3(i=1, 2, 3)) of the pixel P is expressed as the following Expression(7), where output of R at the timing t1 is ζ1, output of G at the timingt2 is ζ2, output of B at the timing t3 is ζ3, spectra of R, G, B thatare perceivable by a human eye are R(λ), G(λ), and B(λ), respectively.The color images at the timings t1 to t3 may be generated based on thisrelationship.A(λ)=ζ₁ R(λ)+ζ₂ G(λ)+ζ₃ B(λ)  (7)

R(λ): an incident light spectrum of R at the timing t1

G(λ): an incident light spectrum of G at the timing t2

B(λ): an incident light spectrum of B at the timing t3

A(λ): an incident light spectrum at the timing t (=(t1+t2+t3)/3)

In such a manner, in the present embodiment, in both cases of shooting astill image and of shooting a moving image, color images are allowed tobe generated successively. It is to be noted that the image processingsection 22 may also use, for example, a color interpolation processingsuch as a demosaic processing other than the above-described arithmeticprocessing. Also, the image processing section 22 may separatelyperform, for example, a white balance adjusting processing, a gammacorrection processing, etc.

As described above, in the image pickup unit 1 of the presentembodiment, it's the transmission wavelength (for example, R, G, and B)of the variable filter 20 is switched time-divisionally in the variablefilter 20. Accordingly, as the image pickup data D0, one-color pixeldata of R, G, and B corresponding to the transmission wavelengths of thevariable filter 20 are allowed to be acquired in a temporally-successivemanner.

Here, in a typical image pickup unit that performs color photography, acolor filter (a filter in which respective transmission regions for R,G, and B are arranged space-divisionally) that transmits one color of R,G, and B is provided on the light receiving face of the image sensor. Insuch a configuration, only pixel data of one color of R, G, or B isobtained in each pixel in the image sensor. Therefore, it is necessaryto perform a color interpolation processing to generate the pixel dataof R, G, and B of the number of the pixels in the image sensor. In suchan image pickup unit, in a case where a color image having a largecontrast difference is generated in particular, a so-called false color,color mixture, etc. may be easily caused, and therefore, display qualityis degraded.

On the other hand, in the present embodiment, as described above, thevariable filter 20 is used to acquire the respective pixel data of R, G,and B in a temporally-successive manner, and the predetermined imageprocessing is performed based on these pixel data of R, G, and B togenerate the color images at the respective shooting timings. Timings ofacquiring images of R, G, and B by time-divisional drive are performedin an extremely-short period. Therefore, the respective images of R, G,and B acquired in a temporally-successive manner are acquiredapproximately at the same time. Therefore, even when the respectiveimages of R, G, and B are summed up, and an arithmetic processing foraveraging is performed, false color, color mixture, etc. are less likelyto be caused. Therefore, it is possible to suppress occurrence of falsecolor, color mixture, etc. and to acquire a color image with high imagequality.

Hereinafter, modifications (Modifications 1 to 4) of the above-describedembodiment will be described. It is to be noted that the same numeralsare used to designate components similar to those in the image pickupunit 1 of the above-described embodiment, and the description thereofwill be appropriately omitted.

[Modification 1]

In the above-described embodiment, description has been given on thecase in which the image processing section 22 performs the arithmeticprocessing to generate the color image using the time-series data of R,G, and B. However, at the time of generating the color image, data ofanother wavelength, for example, of a near-infrared ray (IR), etc. maybe further used in addition to R, G, and B.

However, in the present modification, it may be desirable to use, forthe photodiode 111A arranged in each pixel P in the image sensor 10, aphotoelectric conversion material that has favorable sensitivity withrespect to near-infrared rays (for example, 0.7 μm to 2.5 μm),middle-infrared rays (for example, 2.5 μm to 4 μm), or far-infrared rays(for example, 4 μm to 20 μm) in addition to the respective wavelengthsof R, G, and B. As such a photoelectric conversion material, forexample, compound semiconductors can be mentioned such as PbTe, SnTe,PbSe, and SnSe, as well as PbSnTe, PbSnSeTe, and PbEuTe that are mixedcrystal thereof. These compound semiconductors have an extremely-narrowband gap (0.3 eV or smaller), and therefore, have favorable absorptionrate of far-infrared rays and light having a shorter wavelength (awavelength of 20 μm or shorter). Therefore, due to wavelength dependencyof an absorption coefficient of the material, photoelectric conversionis allowed to be performed on arbitrary wavelength in a range from aultraviolet range to an infrared range. It is to be noted that thephotoelectric conversion material in the present modification is notlimited to such a compound semiconductor, and an organic semiconductorthat serves as a photoelectric conversion material may be used. It is tobe noted that it goes without saying that such compound semiconductors,such organic semiconductors, etc. that have sensitivity covering theinfrared range are allowed to be used also in a case where the colorimage is generated using only the data of R, G, and B as in theabove-described embodiment.

On the other hand, a variable filter similar to that in theabove-described embodiment may be used as the variable filter 20. Forexample, a Lyot filter configured by laminating the liquid crystal cells20A to 20C as in the above-described embodiment may be used. In thepresent modification, it is possible to drive the variable filter 20 toallow near-infrared rays to be selectively transmitted at a certaintiming by appropriately setting the combination of the drive voltages φ1to φ3 for the respective liquid crystal cells 20A to 20C.

In the present embodiment, while the variable filter 20 is used totransmit four wavelengths of R, G, B, and IR by time-divisionalswitching, image pickup is performed, in each pixel P in the imagesensor 10, using the above-described photodiode 111A that hassensitivity to the infrared range. Accordingly, in each pixel P,respective pixel data Dr, Dg, Db, and Di of R, G, B, and IR are acquiredsuccessively in time series. The image pickup data D0 that includes thetime-series data of R, G, B, and IR acquired in such a manner isoutputted to the image processing section 22.

FIG. 12 is a flow chart diagram of an image arithmetic processingaccording to Modification 1. In the present modification, the imageprocessing section 22 generates a color image as follows. Specifically,first, the image processing section 22 acquires, as the image pickupdata D0, the respective pixel data Dr(t1), Dg(t2), Db(t3), and Di(t4) ofR, G, B, and IR obtained at the successive timings t1 to t4, based onthe control by the system control section 16.

Subsequently, the image processing section 22 performs registration, forexample, by the above-described optical flow method, with the use of theacquired pixel data Dr(t1), Dg(t2), Db(t3), and Di(t4), to generatepixel data of three colors of R, G, and B at the timing t1 (step S5).Specifically, in a manner similar to that in the above-describedembodiment, the movement amount (u12, v12) between the timing t at whichthe R pixel data Dr(t1) is acquired and the timing t2 at which the Gpixel data Dg(t2) is acquired is calculated (step S51). Taking intoconsideration the movement amount (u12, v12) calculated in such amanner, G pixel data Drg(t1) at the timing t1 is estimated (step S52).Further, the movement amount (u13, v13) between the timing t at whichthe R pixel data Dr(t1) is acquired and the timing t3 at which the Bpixel data Db(t3) is acquired is calculated (step S53). Taking intoconsideration the movement amount (u13, v13) calculated in such amanner, B pixel data Drb(t1) at the timing t1 is estimated (step S54).Further, in the present modification, a movement amount (u14, v14)between the timing t1 at which the R pixel data Dr(t1) is acquired andthe timing t4 at which the IR pixel data Di(t4) is acquired iscalculated (step S55). Taking into consideration the movement amount(u14, v14) calculated in such a manner, the IR pixel data Dri(t1) at thetiming t1 is estimated (step S56).

Subsequently, the image processing section 22 adds the G pixel dataDrg(t1), the B pixel data Drb(t1), and the IR pixel data Dri(t1) at thetiming t1 determined by the above-described estimation arithmeticprocessing to the R pixel data Dr(t1) acquired at the timing t1. Theimage processing section 22 thereby generates color (thee-primary-color)pixel data (data for one pixel) Drgb(t1) at the timing t1. Such anarithmetic processing is performed on all of the pixels P and theresults are summed up to generate the color image Irgb(t1). Further,pixel data are acquired in a temporally-successive manner repeatedly inorder of R→G→B→IR also for a timing t5 and subsequent timings, andsimilar arithmetic processings are performed using the respectivetimings t2, t3, . . . , as references. Thus, color images Irgb(t2),Irgb(t3), . . . are allowed to be generated also for the timing t2 andthe subsequent timings.

As in the present modification, the pixel data based on thenear-infrared rays may be acquired in addition to those based on therespective color light of R, G, and B, and these pixel data for the fourwavelengths may be used to generate the color images. Also in this case,as in the above-described embodiment, the color image is generated basedon the time-series data of R, G, and B. Therefore, it is possible tosuppress occurrence of false color, color mixture, etc. Accordingly, itis possible to obtain an effect equivalent to that in theabove-described embodiment.

Further, by using the near-infrared rays, it is possible to acquire abright image, for example, even in a dark place. Further, here, when atypical color filter is used, it is necessary to provide an IR filter byspace division. Therefore, it is not allowed to apply an existing colorarrangement (such as the Bayer arrangement) as it is. In the presentmodification, as described above, it is possible to switch thetransmission wavelength to the near-infrared ray by electric control ofthe variable filter 20, and therefore, change in design of wavelength iseasy.

[Modification 2]

FIG. 13 schematically illustrates part of a cross-sectional structure ofan image pickup unit that includes an image sensor (an image sensor 10B)according to Modification 2. Also in the present modification, thevariable filter 20 is provided on a light receiving face of the imagesensor 10B. However, in the present modification, the image sensor 10Bmay be, for example, a CMOS of a so-called front illumination type, thatincludes a device layer 116 including the photoelectric conversion layer114 on the substrate 11, and in a layer located in a lower positioncompared to the wiring layer 112. Also in this structure, as in theabove-described case of the back illumination type, the above-describedtransistors Tr1 to Tr4 for pixel driving and the transistors Tr5 to Tr7configuring part of the wavelength selection circuit 21 are formed inthe device layer 116 together with the photoelectric conversion layer114. However, in the present modification, a waveguide layer 118 isformed in a region corresponding to the wiring layer 117 directly on thephotoelectric conversion layer 114. The on-chip lens 115 is provided onthis waveguide layer 118. In such a manner, the image sensor 10B is notlimited to that of the back illumination type, and that of the frontillumination type may be used as the image sensor 10B.

[Modification 3]

FIG. 14 illustrates a cross-sectional structure of a variable filter (avariable filter 30) according to Modification 3. As with the variablefilter 20 in the above-described embodiment, the variable filter 30 inthe present modification is provided on the light receiving face of theimage sensor 10, and has the function of time-divisionally switching thetransmission wavelength. Further, the variable filter 30 may beconfigured, for example, of a liquid crystal Lyot filter in which aplurality of liquid crystal cells are laminated. The variable filter 30may include a plurality of cell units U in a direction along thesubstrate surface as necessary.

However, in the present modification, the variable filter 30 isconfigured by alternately laminating seven (seven stages of) liquidcrystal cells 20A to 20G, and eight polarizing plates 22 a to 22 i. Eachof these seven liquid crystal cells 20A to 20G has a configurationsimilar to that of the liquid crystal cells 20A to 20C described in theabove embodiment. Each of these seven liquid crystal cells 20A to 20G iselectrically connected to a transistor (not illustrated in FIG. 14)arranged in the image sensor 10 via the anisotropic conductive film 210and the connection bump 211. It is to be noted that FIG. 14 illustratesonly part of the through-hole vias for connecting the respective liquidcrystal cells 20A to 20G to the transistors, the anisotropic conductivefilm 210, and the connection bump 211. These vias, the anisotropicconductive film 210, and the connection bump 211 are formed also inother portions which are not illustrated. Accordingly, the liquidcrystal cells 20A to 20G are allowed to be electrically controlled in aseparated manner.

In such a manner, also in the case in which the number of stages in thevariable filter 30 is increased, and for example, seven liquid crystalcells 20A to 20G are laminated, by appropriately setting the combinationof the drive voltages φ1 to φ7 for the respective liquid crystal cells20A to 20G, it is possible to transmit a desirable selective wavelength,and to time-divisionally switch its transmission wavelength, as in theabove-described embodiment.

Moreover, as in the present modification, by increasing the number ofstages of the liquid crystal cells, it is possible to control thespectrum of the transmission wavelength of the variable filter moreprecisely. As an example, FIG. 15 illustrates respective transmissionwavelengths λa to λg of the liquid crystal cells 20A to 20G, and atransmission wavelength λ₀ of the variable filter 30 as a whole. In thisexample, by setting the respective transmission wavelengths λa to λg ofthe liquid crystal cells 20A to 20G as shown in the drawing, it ispossible to set a spectrum peak of the transmission wavelength at about500 nm. Further, it is possible to narrow FHM to about 20 nm, andthereby, in particular, absorption or reflective spectrum intensityunique to a substance is allowed to be easily acquired. In other words,concentration, position, etc. of a specific substance is allowed to bedetected. Therefore, an image pickup unit suitable for an electronicapparatus in which a measurement function, a detection function, etc. ofa specific substance are necessary.

[Modification 4]

FIG. 16 illustrates an outline configuration of a variable filter (avariable filter 40) according to Modification 4. In all of theabove-described embodiment and Modifications 1 to 3, the liquid crystalLyot filter has been mentioned as an example of the variable filter.However, a piezoelectric Fabry-Perot interferometer may be used as inthe present modification. In this piezoelectric Fabry-Perotinterferometer, for example, a piezoelectric device 42 is provided as aspacer between a pair of substrates 41 a and 41 b that face each other.By driving the piezoelectric device 42 with a voltage, a cell gapbetween the substrates 41 a and 41 b is allowed to be adjusted.Semi-transmissive mirrors 43 a and 43 b that each reflect or transmitlight are attached to the respective facing faces of the substrates 41 aand 41 b. In such a configuration, incident light L is interfered bybeing reflected between the substrates 41 a and 41 b (between thesemi-transmissive mirrors 43 a and 43 b), and thereby, only a selectivewavelength is transmitted. At this time, by adjusting the cell gap bycontrol of the piezoelectric device 42, it is possible to set thetransmission wavelength arbitrarily. In such a manner, the variablefilter 40 may be configured of the piezoelectric Fabry-Perotinterferometer. Also in this case, the cell gap is varied by electricand mechanical control, and thereby, the transmission wavelength isswitched. Therefore, it is possible to perform time-divisional switchingbetween the respective wavelengths of R, G, and B. Accordingly, it ispossible to acquire the time-series data of R, G, and B to generate thecolor image as in the above-described embodiment. Therefore, it ispossible to obtain an effect equivalent to that in the above-describedembodiment.

Application Examples

The image pickup unit 1 including the configuration according to theabove-described embodiment and Modifications 1 to 4 is applicable tovarious types of electronic apparatuses that have a shooting function, ameasurement function, a display function, etc. As described above, theimage pickup unit 1 is allowed to provide a color image with high imagequality. Therefore, the image pickup unit 1 is suitable for a camera (adigital still camera or a video camcorder), a mobile apparatuses such asa mobile phone or a PDA (Personal Digital Assistant) having a shootingfunction, etc. Further, in addition thereto, the image pickup unit 1 isapplicable to a measurement (detection) apparatus for a specificsubstance, etc. FIG. 17 illustrates a functional block configuration ofa camera (a camera 2) as an example thereof.

The camera 2 includes an optical system including a lens group 31, etc.,the image pickup unit 1, a DSP circuit 32 that is a camera signalprocessing section, a frame memory 35, a display unit 33, a recordingunit 36, an operation system 34, a power source system 37, etc. Out ofthese, the DSP circuit 32, the frame memory 35, the display unit 33, therecording unit 36, the operation system 34, and the power source system37 are configured to be connected to one another via a bus line 38.

The lens group 31 takes in incident light (image light) from a subject,and to form an image on an image pickup plane (the light receiving face)of the image pickup unit 1. The lens group 31 is configured of one or aplurality of lenses. The image pickup unit 1 outputs the image pickupdata D0 based on the incident light that has been formed into an imageon the image pickup plane by the lens group 31. The display unit 33 maybe configured, for example, of a liquid crystal display unit, an organicEL (electro luminescence) display unit, or the like. The display unit 33displays a moving image or a still image (a color image after the imageprocessing performed by the image processing section 22) shot by theimage pickup unit 1. The recording unit 36 records the moving image orthe still image shot by the image pickup unit 1 on a recording mediumsuch as a video tape and a DVD (Digital Versatile Disk). The operationsystem 34 serves as an external signal input means in accordance withoperation by a user. The operation system 34 receives an operationinstruction on various functions of the camera 2, and to transmit thereceived operation instruction to inside. The power source system 37includes various power sources that serve as operation power source ofthe DSP circuit 32, the frame memory 35, the display unit 33, therecording unit 36, and the operation system 34.

Hereinabove, description has been given referring to the embodiment andthe modifications. However, the content of the present disclosure is notlimited to the above-described embodiment, and may be variouslymodified. For example, in the above-described embodiment and the like,the CMOS of the back illumination type or the front illumination typehas been mentioned as an example of the image sensor. However, the imagesensor is not limited to the CMOS, and may be a CCD (Charge CoupledDevice Image Sensor) or a MOS-type image sensor.

Moreover, in the above-described embodiment and the like, descriptionhas been given on the case in which time-divisional switching betweenthree wavelengths of R, G, and B or four wavelengths of R, G, B, and IRis performed in the variable filter to transmit them. However, thetransmission wavelength of the variable filter is not limited thereto,and an arbitrary wavelength to be necessary is allowed to be set. Forexample, other than R, G, and B, a wavelength of yellow (for example,from 570 nm to 590 nm) or of orange (for example, from 590 nm to 620 nm)may be transmitted. Further, when the image pickup unit 1 is used for ameasurement (detection) apparatus for a specific substance, a wavelengthcorresponding to a substance to be a target of measurement may betransmitted. As described above, in the variable filter, it is possibleto adjust its transmission wavelength by electric control (or mechanicalcontrol). Therefore, the transmission wavelength is allowed to be set toa desirable transmission wavelength as necessary. Moreover, the numberof wavelength for time-divisional switching is not limited to threewavelengths or four wavelengths described above, and may be twowavelengths, five wavelengths, or more.

It is to be noted that the present disclosure may have configurationsdescribed in (1) to (14) below.

(1) An image pickup unit including:

an image pickup device including a plurality of pixels and outputting animage pickup data;

a variable filter provided on a light receiving face of the image pickupdevice, and configured to allow a selective transmission wavelength ofincident light to be variable; and

a filter drive section driving the variable filter and therebytime-divisionally switching the transmission wavelength.

(2) The image pickup unit according to the above-described (1), wherein

the plurality of pixels in the image pickup device are arrangedtwo-dimensionally,

the variable filter includes a plurality of sub-filters each facing apixel in the image pickup device, and

the filter drive section separately drives the plurality of sub-filters.

(3) The image pickup unit according to the above-described (1), wherein

the plurality of pixels in the image pickup device are arrangedtwo-dimensionally,

the variable filter includes a plurality of sub-filters each facing apixel column or a pixel row in the image pickup device, and

the filter drive section separately drives the plurality of sub-filters.

(4) The image pickup unit according to the above-described (1), wherein

the variable filter is provided integrally over all of the pixels in theimage pickup device.

(5) The image pickup unit according to any one of the above-described(1) to (4), wherein

the filter drive section drives the variable filter to allow respectivecolor light of R (red), G (green), and B (blue) to be transmittedtime-divisionally.

(6) The image pickup unit according to the above-described (5), furtherincluding

an image processing section using pixel data of respective colors of R,G, and B, as the image pickup data outputted from the image pickupdevice at temporally-successive timings, to generate color images atrespective timings.

(7) The image pickup unit according to the above-described (6), wherein

the image processing section calculates respective movement amounts of asubject between first pixel data of a wavelength of one of R, G, and Band second and third pixel data, the first pixel data being acquired ata first timing, and the second and third pixel data being acquired atsecond and third timings temporally subsequent to the first timing,respectively,

the image processing section generates pixel data of a wavelengthcorresponding to the second pixel data at the first timing, based on themovement amount of the subject between the first and the second pixeldata, and

the image processing section generates pixel data of a wavelengthcorresponding to the third pixel data at the first timing, based on themovement amount of the subject between the first and the third pixeldata.

(8) The image pickup unit according to the above-described (5), wherein

the filter drive section drives the variable filter to allow therespective color light of R, G, and B and a near-infrared ray to betransmitted time-divisionally.

(9) The image pickup unit according to the above-described (8), furtherincluding

an image processing section using pixel data of respective wavelengthsof R, G, and B and of the near-infrared ray, as the image pickup dataoutputted from the image pickup device at temporally-successive timings,to generate color images at respective timings.

(10) The image pickup unit according to the above-described (9), wherein

the image processing section calculates respective movement amounts of asubject between first pixel data corresponding to a wavelength of one ofR, G, B and a near-infrared ray, and second to fourth pixel data, thefirst pixel data being acquired at a first timing, and the second tofourth pixel data being acquired at second to fourth timings temporallysubsequent to the first timing, respectively,

the image processing section generates pixel data of a wavelengthcorresponding to the second pixel data at the first timing, based on themovement amount of the subject between the first and the second pixeldata,

the image processing section generates pixel data of a wavelengthcorresponding to the third pixel data at the first timing, based on themovement amount of the subject between the first and the third pixeldata, and

the image processing section generates pixel data of a wavelengthcorresponding to the fourth pixel data at the first timing, based on themovement amount of the subject between the first and the fourth pixeldata.

(11) The image pickup unit according to the above-described (1) to (10),wherein

the variable filter is configured of a liquid crystal Lyot filter.

(12) The image pickup unit according to the above-described (11),wherein

the liquid crystal Lyot filter is a lamination of a plurality of liquidcrystal cells, and

the filter drive section applies drive voltages to the respectiveplurality of liquid crystal cells, and sets the transmission wavelengthof the liquid crystal Lyot filter based on a combination of those drivevoltages.

(13) The image pickup unit according to any one of the above-described(1) to (10), wherein

the variable filter is configured of a piezoelectric Fabry-Perotinterferometer.

(14) An electronic apparatus with an image pickup unit, the image pickupunit including:

an image pickup device including a plurality of pixels and outputting animage pickup data;

a variable filter provided on a light receiving face of the image pickupdevice, and configured to allow a selective transmission wavelength ofincident light to be variable; and

a filter drive section driving the variable filter and therebytime-divisionally switching the transmission wavelength.

It should be understood that various changes and modifications to thepresently preferred embodiments described herein will be apparent tothose skilled in the art. Such changes and modifications can be madewithout departing from the spirit and scope of the present subjectmatter and without diminishing its intended advantages. It is thereforeintended that such changes and modifications be covered by the appendedclaims.

The invention is claimed as follows:
 1. An image pickup unit comprising:an image pickup device including a plurality of pixels; a variablefilter provided on a light receiving face of the image pickup device,the variable filter is configured to selectively transmit incidentlight; a filter drive section configured to control the variable filterand control transmission wavelength of the variable filter, the firstdrive section includes: a digital potentiometer configured to output asignal voltage; an operation amplifier connected to the digitalpotentiometer; a transistor connected to the variable filter; wherein afirst terminal of the transistor is connected to a selection line thatis configured to provide a control signal, wherein a second terminal ofthe transistor is connected to an output terminal of the operationamplifier, and wherein a third terminal is connected to the variablefilter.
 2. The image pickup unit according to claim 1, wherein the firstterminal is a gate and the second terminal is a drain or a source. 3.The image pickup unit according to claim 1, wherein the pixels arearranged two-dimensionally, wherein the variable filter includes aplurality of sub-filters facing a pixel in the image pickup device, andwherein the filter drive section is configured to control thesub-filters.
 4. The image pickup unit according to claim 1, wherein thevariable filter is provided integrally over all of the pixels.
 5. Theimage pickup unit according to claim 1, wherein the pixels are arrangedtwo-dimensionally, wherein the variable filter includes a plurality ofsub-filters facing at least one of a pixel column and a pixel row in theimage pickup device, and wherein the filter drive section is configuredto control the sub-filters.
 6. The image pickup unit according to claim1, wherein the filter drive section is configured to control thevariable filter to allow light including Red, Green, Blue, andnear-infrared light to be transmitted time-divisionally.
 7. The imagepickup unit according to claim 6, wherein the image processing sectionis configured to use pixel data of the light as the image pickup data togenerate color images at respective timings.
 8. The image pickup unitaccording to claim 1, wherein the variable filter includes a liquidcrystal Lyot filter.
 9. The image pickup unit according to claim 8,wherein the liquid crystal Lyot filter includes a lamination including aplurality of liquid crystal cells, and wherein the filter drive sectionis configured to apply voltages to the plurality of liquid crystalcells, and to set the transmission wavelength of the liquid crystal Lyotfilter based on a combination of the voltages.
 10. The image pickup unitaccording to claim 1, wherein the variable filter includes apiezoelectric Fabry-Perot interferometer.
 11. The image pickup unitaccording to claim 1, wherein the image pickup device includes a backillumination type image sensor.
 12. An electronic apparatus comprising:an image pickup unit, the image pickup unit including: an image pickupdevice including a plurality of pixels; a variable filter provided on alight receiving face of the image pickup device, the variable filter isconfigured to selectively transmit incident light; a filter drivesection configured to control the variable filter and controltransmission wavelength of the variable filter, the first drive sectionincludes: a digital potentiometer configured to output a signal voltage;an operation amplifier connected to the digital potentiometer; atransistor connected to the variable filter; wherein a first terminal ofthe transistor is connected to a selection line that is configured toprovide a control signal, wherein a second terminal of the transistor isconnected to an output terminal of the operation amplifier, and whereina third terminal is connected to the variable filter.
 13. The electronicapparatus according to claim 12, wherein the first terminal is a gateand the second terminal is a drain or a source.
 14. The electronicapparatus according to claim 12, wherein the pixels are arrangedtwo-dimensionally, wherein the variable filter includes a plurality ofsub-filters facing a pixel in the image pickup device, and wherein thefilter drive section is configured to control the sub-filters.
 15. Theelectronic apparatus according to claim 12, wherein the variable filteris provided integrally over all of the pixels.
 16. The electronicapparatus according to claim 12, wherein the pixels are arrangedtwo-dimensionally, wherein the variable filter includes a plurality ofsub-filters facing at least one of a pixel column and a pixel row in theimage pickup device, and wherein the filter drive section is configuredto control the sub-filters.
 17. The electronic apparatus according toclaim 12, wherein the filter drive section is configured to control thevariable filter to allow light including Red, Green, Blue, andnear-infrared light to be transmitted time-divisionally.
 18. Theelectronic apparatus according to claim 12, wherein the variable filterincludes a liquid crystal Lyot filter.
 19. The electronic apparatusaccording to claim 18, wherein the liquid crystal Lyot filter includes alamination including a plurality of liquid crystal cells, and whereinthe filter drive section is configured to apply voltages to theplurality of liquid crystal cells, and to set the transmissionwavelength of the liquid crystal Lyot filter based on a combination ofthe voltages.
 20. The electronic apparatus according to claim 12,wherein the image pickup device includes a back illumination type imagesensor.